System and method for providing a dynamically reconfigurable integrated virtual environment

ABSTRACT

The present disclosure relates to a system, comprising: a first server computing device configured to store various application data relating to the system, and control a first plurality of modules to simultaneously establish multiple logically separate and secure networks within a self-supported computing environment; a second server computing device configured to control a second plurality of modules to perform out-of-band management of the system; and a third server computing device configured to control a third plurality of modules to control inbound and outbound data traffic of the logically separate and secure networks. The system is scalable by at least adding additional one or more first server computing devices to host additional application data within the self-supported computing environment’s secure configuration and logical separation of networks while maintaining the second and third server computing devices.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. Pat. ApplicationNo. 17/119,394 filed on Dec. 11, 2020, which claims priority to U.S.Provisional Pat. Application No. 62/947,336 filed on Dec. 12, 2019,entitled “SYSTEM AND METHOD FOR PROVIDING A DYNAMICALLY RECONFIGURABLEINTEGRATED VIRTUAL ENVIRONMENT,” the contents of which are incorporatedby reference herein in their entirety.

FIELD OF TECHNOLOGY

The present disclosure generally relates to systems and methods forproviding a dynamically reconfigurable integrated virtual computingenvironment or platform, and more particularly relates to aconfiguration of a hardware platform for expanding multiple securevirtual networks hosting capabilities across a converged infrastructuremanaged systems and subsystems to provide information sharingcapabilities among unilateral, bilateral and multilateral stakeholders.

BACKGROUND

Networks involving securely exchanging information among multipleseparate bilateral and/or multilateral external stakeholders via, e.g.,data separation has historically been limited to physically separatinghardware components for validating information separation. Cloudcomputing efficiencies, within software defined networks and softwaredefined data centers, are therefore not realized to the maximum extentpracticable due to physical separation requirements for securecryptographic network information sharing. Often, policies requirephysical separation based on the sensitivity levels specified by thegoverning organization.

Accordingly, there is a need for enabling multiple networks to beconfigured and dynamically re-provisioned on a single platformenvironment to support autonomous and simultaneous network hosting withsecure cryptographic separation of data.

SUMMARY

Among other features, the present disclosure provides a system,comprising: a first server computing device, comprising: a firstnon-transitory computer-readable storage medium configured to store afirst set of instructions and application data relating to the system,and a first processor coupled to the first non-transitorycomputer-readable storage medium and configured to control a firstplurality of modules to execute the first set of instructions forsimultaneously establishing a plurality of logically separate and securenetworks within a self-supported computing environment; a second servercomputing device, comprising: a second non-transitory computer-readablestorage medium configured to store a second set of instructions, and asecond processor coupled to the second non-transitory computer-readablestorage medium and configured to control a second plurality of modulesto execute the second set of instructions for performing out-of-bandmanagement of the system; and a third server computing device,comprising: a third non-transitory computer-readable storage mediumconfigured to store a third set of instructions, and a third processorcoupled to the third non-transitory computer-readable storage medium andconfigured to control a third plurality of modules to execute the thirdset of instructions for controlling inbound and outbound data traffic ofthe plurality of logically separate and secure networks, wherein thesystem is scalable by at least adding additional one or more firstserver computing devices to host additional application data within theself-supported computing environment’s secure configuration and logicalseparation of networks while maintaining the second and third servercomputing devices.

In one embodiment, the present disclosure relates to a non-transitorycomputer-readable storage medium having instructions embodied thereinthat when executed cause a computer system to perform: storing a firstset of instructions and application data relating to the computer systemon a first non-transitory computer-readable storage medium of a firstserver computing device of the computer system, executing the first setof instructions, by a first processor of the first server computingdevice coupled to the first non-transitory computer-readable storagemedium, to simultaneously establish a plurality of logically separateand secure networks within a self-supported computing environment;storing a second set of instructions on a second non-transitorycomputer-readable storage medium of a second server computing device ofthe computer system; executing the second set of instructions, by asecond processor of the second server computing device coupled to thesecond non-transitory computer-readable storage medium, to performout-of-band management of the system; storing a third set ofinstructions on a third non-transitory computer-readable storage mediumof a third server computing device of the computer system; executing thethird set of instructions, by a third processor of the third servercomputing device coupled to the third non-transitory computer-readablestorage medium, to control inbound and outbound data traffic of theplurality of logically separate and secure networks; and addingadditional first server computing devices to the computer system to hostadditional application data within the self-supported computingenvironment while maintaining the second and third server computingdevices.

The above simplified summary of example aspects serves to provide abasic understanding of the present disclosure. This summary is not anextensive overview of all contemplated aspects, and is intended toneither identify key or critical elements of all aspects nor delineatethe scope of any or all aspects of the present disclosure. Its solepurpose is to present one or more aspects in a simplified form as aprelude to the more detailed description of the disclosure that follows.To the accomplishment of the foregoing, the one or more aspects of thepresent disclosure include the features described and exemplary pointedout in the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated into and constitute apart of this specification, illustrate one or more example aspects ofthe present disclosure and, together with the detailed description,serve to explain their principles and implementations.

FIG. 1 illustrates a block diagram of a computing platform, according toan exemplary aspect of the present disclosure;

FIG. 2 illustrates management components of the computing platform andthat management components are physically separated from datacomponents, according to an exemplary aspect of the present disclosure.

FIG. 3 illustrates a high-level dedicated encrypted tunnel concept,according to an exemplary aspect of the present disclosure;

FIG. 4 illustrates an expanded horizontal view of virtual machine (VM)East /West traffic, according to an exemplary aspect of the presentdisclosure;

FIG. 5 illustrates additional physical protections for the East/Westtraffic across a switch, according to an exemplary aspect of the presentdisclosure;

FIG. 6 illustrates a representative model of the Edge server’s externalconnections and the external networks, according to an exemplary aspectof the present disclosure;

FIG. 7 illustrates an overview of hypervisor VM encryption, according toan exemplary aspect of the present disclosure;

FIG. 8 illustrates various filters of a VM layer, according to anexemplary aspect of the present disclosure;

FIG. 9 illustrates a simplified topology of an external key managementsystem (KMS), according to an exemplary aspect of the presentdisclosure;

FIG. 10 illustrates a high-level overview of unified extensible firmwareinterface (UEFI) secure boot VIB certification process, according to anexemplary aspect of the present disclosure;

FIG. 11 illustrates an architecture of role based access control (RBAC)rights management tools, according to an exemplary aspect of the presentdisclosure;

FIG. 12 illustrates an example of hyper-converged infrastructure (HCI)storage, according to an exemplary aspect of the present disclosure; and

FIG. 13 illustrates three software defined networking (SDN) planes,according to an exemplary aspect of the present disclosure.

DETAILED DESCRIPTION

Various aspects of invention will be described with reference to thedrawings, wherein like reference numerals are used to refer to likeelements throughout. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to promotea thorough understanding of one or more aspects of the invention. It maybe evident in some or all instances, however, that any aspects describedbelow can be practiced without adopting the specific design detailsdescribed below.

Recent technology advancement to combine network connectivity, computecapacity, memory and file storage across common infrastructure hasresulted in convergence of information technology hardware to reducesize, weight and power across components required to support informationsharing systems. For example, a hardware approach to software definednetworks generally requires utilizing network virtualizationtechnologies such as a bare metal hypervisor to share memory, processingand storage resources across physically separate hardware components.

When running a single physical network, efficiencies may be securelyleveraged across separate hardware components residing within the samenetwork. Complexities due to secure information separation may beintroduced if more than one network is hosted and accessible via certainhyper-converged infrastructure components (e.g., HCI servers orcomputing devices) as each network would have access to all hardwaredevices that are coupled together. It is undesirable to configure, hostand maintain multiple infrastructure networks across physically separatehardware due to the inability to efficiently provision new networkswithout additional hardware. Networking, computing and storage resourcesare generally not fully allocated for each system to ensure efficientprocessing utilization for user operations which creates a pool ofexcess overall system resources that are not realized due to physicalcryptographic system constraints with multiple networks that may requirephysical separation.

Accordingly, it is desirable to validate and maintain secure network anddata separation within a hyper-converged environment in a logical andvirtual configuration to provide the same level of security as if thesystem being deployed contained physically separate hardware componentswithout the loss of unutilized processing, memory and storage resourcepools.

On a high level, the present disclosure provides a secure configurationand integration of hardware and software components to utilize totalsystem resources to operate multiple simultaneous virtualized networks;and the ability to pre-stage networks for dynamic provisioning andswapping (e.g., hot swapping which may relate to the replacement oraddition of components to a computing system without stopping, shuttingdown, or rebooting the computing system) of hosted active and inactivenetworks. Aspects of the present disclosure may relate to ensuringsecure cryptographic separation of data within multiple simultaneousnetworks within one hyper-converged infrastructure environment.

As will be described fully below, the present disclosure generallyrelates to a pre-integrated computing platform configured to providemulti-tenant isolation of virtualized networks using software definednetworking technologies and a hyper-converged infrastructure to realizesecure network virtualization. The disclosed computing platform may beconfigured to host multiple virtualized networks data centers on asingle platform and provide consolidated data center services applicableto industry best practices and cybersecurity risk thresholds for suchvirtualized networks. One of the primary objectives of the disclosedcomputing platform may include improving system performance, reducingthe time needed to swap out (turn on turn off) virtualized networks, andproviding for rapid expansion, modification, or depi-ecation ofvirtualized networks.

FIG. 1 is a block diagram of a computing platform 100, according toaspects of the present disclosure. In one embodiment, computing platform100 may include one or more management servers 102, one or moreapplication servers 104, and one or more edge servers 106. Among otherfeatures, computing platform 100 may be configured to pre-stage existingcomputing systems (e.g., a portion of computing systems 1-n) based onknown standards and specifications, create a new computing system (e.g.,one of computing systems 1-n) based on pre-positioned templates, hostmultiple concurrent operational computing systems, ensure a secureseparation of virtualized networks, host stand-by computing systems(e.g., a portion of computing systems 1-n), and dynamically swap betweencomputing systems that are active or in stand-by. Such computing systemsmay be scalable by at least adding additional application servers 104 tohost additional application data within a self-supported computingenvironment’s secure configuration and logical separation of networkswhile maintaining management servers 102 and edge servers 106. Aself-supported computing environment may include the ability for theentirety of such computing systems to run without needing any outsideconnectivity for services. All resources necessary to function may becompletely contained within the physical and virtual infrastructure ofsuch computing systems. Each computing system 1-n and computing platform100 may be communicatively coupled over a network (e.g., Ethernet), suchthat each computing system 1-n may be configured to access functionalityand/or use some of resources of a virtualized environment provided bycomputing platform 100,

Each management server 102, application server 104, and edge server 106may be constructed on a hardware platform (not shown) including one ormore central processing units (CPUs), system memory, and storage. Thehardware platform may also include one or more network interfacecontrollers (NICs) that connect each server 102, 104, 106 to a network,and one or more host bus adapters (HBAs) that connect each server 102,104, 106 to a persistent storage unit. Further, for each management,application, and edge servers 102, 104, 106, virtualization software, orfirmware or hardware 102 a, 104 a, 106 a may be respectively installedon top of the hardware platform of each server and support a VMexecution space within which one or more virtual computing systems 1-nmay be concurrently instantiated and executed. In one embodiment, eachvirtual computing systems 1-n may be configured to implement a virtualhardware platform that supports the installation of a guest operatingsystem (OS) for at least executing various applications. In someembodiments, each virtualization software 102 a, 104 a, 106 a mayinclude a virtualization kernel component controlling a plurality ofvirtualization sub-components. Each virtualization sub-component maycorrespond to each of a plurality of instantiated VMs. Alterntatively,each virtualization sub-component may be considered to be a part of itscorresponding VM as each virtualization sub-component may includehardware emulation modules for its corresponding VM. It should beappreciated that techniques described herein may be applicable to hostedvirtualized computing systems 1-n or certain non-virtualized computersystems. Further, various computer system configurations for each server102, 104, 106 may be implemented including hand-held devices,microprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, mainframe computers, etc.

Virtualized computing systems 1-n may be logically grouped by any ofserver 102, 104, 106 for a particular workload when an application isexecuted. The computing systems 1-n in each logical group may executeinstructions alone or in combination with one another, for example in adisturbed manner. The virtualization infrastructure established bycomputing platform 100 may also include a plurality of virtualdatacenters (not shown). A virtual datacenter may refer to an abstractpool of resources (e.g., memory, CPU, storage), and a virtual datacenter may be implemented on one or some combination of physicaldevices.

In some embodiments, computing systems 1-n may be deployed in a cloudenvironment, built upon a virtualized environment provided by computingplatform 100. As shown in FIG. 1 , each computing systems 1-n may belocated in an Internet connected data centers 1-n or a private cloudcomputing center coupled with one or more public and/or privatenetworks. Devices and users in each computing system 1-n, in oneembodiment, may couple with a virtual or physical entity through anetwork connection which may be a public network connection, privatenetwork connection, or some combination thereof. For example, a user mayaccess a web page or application presented by one of computing systems1-n at a workstation which may be a virtual or physical entity.

One or more embodiments of the present disclosure may be implemented asone or more computer programs or as one or more computer program modulesembodied in one or more computer readable media. The term “computerreadable medium” refers to any suitable data storage device that canstore data which can thereafter be input to a computer system. Computerreadable media may be based on any existing or subsequently developedtechnology for embodying computer programs in a manner that enables themto be read and executed by a computer. Example computer readable mediummay include a hard drive, network attached storage, read-only memory,random-access memory (e.g., a flash memory device), a compact discs,CD-ROM, a CD-R, or a CD-RW, a digital versatile disc, a magnetic tape,and other optical and non-optical data storage devices. In oneembodiment, the computer readable medium may also be distributed over anetwork coupled computer system so that the computer readable code isstored and executed in a distributed fashion.

In accordance with aspects of the present disclosure, each managementserver 102, application server 104, and edge server 106 of computingplatform 100 may be a hyper-converged computing device includingpre-tested, pre-configured and pre-integrated storage and networkcomponents located in an enclosure. Each server 102, 104, and 106 may bescaled to add more computing nodes, each node including a CPU, memoryand storage. In some embodiments, each computing node may be consideredas a server and configured to independently host a number of virtualizedcomputing systems 1-n. All computing nodes in each server 102, 104, and106 may be independent of one another, and may not be required to shareany functionality with one another. Each server 102, 104, and 106 mayinclude various external interfaces (e.g., universal serial bus (USB)ports) and at least one graphical user interface module having codes orinstructions stored thereon for creating and managing all components ofeach server 102, 104, and 106.

As shown in FIG. 1 , each virtual computing systems 1-n inside computingplatform 100 may be configured to have its own switching and routingcapabilities unique to services of computing systems 1-n. Physicalaccess to virtual computing systems 1-n may be provided by edge server106. In one embodiment, edge server 106 may be configured to provide aconnection from the multi-tenant computing platform 100 to a separatephysical infrastructure via a single or pair of network cables.

In some embodiments, as shown in FIG. 1 , each of computing systems 1-nmay include various devices/user and a data center. The devices mayinclude any number of physical and/or VMs. The physical and/or VMs mayinclude a variety of applications (e.g., operating systems). Thephysical and/or VMs may have the same installed applications or may havedifferent installed applications or software. The installed software maybe one or more software applications from one or more vendors.

As a high level system design overview, three interlocking layers ofsystem protection may be used to secure computing platform 100: 1)computing platform 100 itself, which includes hardware and softwarecomponents; 2) unique system security and hardening for various systemhardware and software components; and 3) dedicated virtualized encryptedtunnels configured to keep individual virtualized computing systems 1-ncommunications secure and separate.

As will be described fully below and in accordance with importantaspects of the present disclosure, computing platform 100 may include afamily of products and services deployed within a communication networkto provide a stable, secure information processing environment in whichmultiple virtualized systems or networks may be kept completely separatefrom one another. For example, computing platform 100 may be configuredto provide a multi-tenant data center using certain virtualizationsoftware (e.g., virtualization software 102 a, 104 a, and 106 a) at itscore for server virtualization, software defined networkingfunctionality for network virtualization, and a host of other productsand services to ensure a stable and secure platform. By providing localdata center services, end users may be provided with full access to thefunctionality provided by respective virtualized networks while havingno direct access to computing platform 100 itself.

In one embodiment, Ethernets 1-n in FIG. 1 may include a physicallyseparated local area networks (LANs) infrastructure dedicated to each ofa number of active computing systems 1-n for implementing, such as,authentication services, E-mail, chat, and web services. The LAN may beconfigured to maintain a physically separated infrastructure andseparate approval boundaries. Each network’s system administrators andusers may access computing system resources from individual end useraccess devices. In addition to edge server 106 which is configured toprovide external physical communication connections to the virtualizednetworks 1-n, computing platform 100 may include multiple infrastructureswitches for internal platform communication.

Further, computing platform 100 may include dedicated encrypted tunnelshaving a variety of system components configured to provide the abilityto securely host multiple virtualized networks on a singlehardware/software platform. As shown in FIG. 1 , each dedicatedencrypted tunnel may provide a multi-tiered protective boundary aroundnetwork traffic as it traverses the system. As a result, the contentsand the communication of each virtualized networks may be isolated fromall other networks.

FIG. 2 illustrates management components of the computing platform 100of FIG. 1 and that a management switch 201 and management servers 102are physically separated from data switch (e.g., data switch 501 of FIG.5 ) and data host components (e.g., application servers 104 of FIG. 1 ),according to aspects of the present disclosure. In one embodiment,management functions (administration, policies, etc. ) may be performedon management servers 102 in FIG. 1 and not the application servers 104,The management servers 102 may be connected to a physical managementswitch (out-of-band management (OBM)) 201. Access control lists (ACL)rules 202 may be pre-provisioned on the computing platform 100 toestablish exception-only-traffic (e.g., traffic that conforms tosecurity policies and ACL rules) that is aligned with management andcontrol plane data. That is, data traffic that does not meet securityrequirements or pre-provisioned ACL rules may be rejected by default.Only data traffic that is conformed to security policy criteria may beallowed. Physical network management port group 203 may be configured toverify that all virtualized network data in transit is isolated andsecured from all other external domains. The physical management switch201 may be connected to the management servers 102 through physicalcabling 204 that are connected to pre-provisioned ports authorized totransmit system policy verified OBM data traffic. Any other networkcommunication 205 not related to the administration of the system may bedesignated to be pre-provisioned (e.g., storage).

Computing platform 100 may be configured to allow for expansion ofmaximum system capacity to meet evolving mission requirements. In oneaspect, designed for modular scalability, computing platform 100 may beconfigured to add processing, storage, network connectivity, memory, orother capacity over time without altering design components. To increasecapacity, additional hardware may be integrated into the hyper-convergedenvironment while maintaining its core security architecture.

The modularized design of computing platform 100 may be configured toprovide replacement or retirement of parts or systems withoutfundamentally breaking its security posture. Failure of a singlecomponent may not create a security violation or allow for unauthorizeddata transfers, inspections, or data exports. Computing platform 100 hasresiliency without the excess redundancy that may artificially increasethe size, footprint, or size, weight and power (SWaP) limitations of thesystem.

The hardware and software components used to create and secure computingplatform 100 may be selected to ensure at least platform stability,platform security, and platform system separation.

In one important aspect, computing platform 100 maybe contemplated inaccordance with the redundant, always invoked, independentimplementations, non-bypassable (RAIN) security architecturerequirements.

Specifically, with respect to redundancy, security-relevant components,including filters and domain separation, may be invoked twice for eachunique function. Redundancy here may not mean the components beinginvoked are independent implementations. Rather, the components may beinvoked or implemented at multiple layers twice. For example, thedisclosed platform architecture may include redundant and layeredsecurity capabilities to direct data and information flow to passmultiple checkpoints.

Computing platform 100 may be configured to ensure that key securityrelevant components are handled across multiple subsystems. No singlesystem failure can allow a violation event. In other words, traffic maynot pass between networks if one or more components of platform 100 havefailed. Redundancy may be implemented for security-relevant componentsby clustering of hardware for both the management and applicationdomains. Further, dual port NICs may be deployed at edge server 106, andVMs in management server 102 may have primary and secondary VMs, such asRBAC rights management tools, management, and an SDN cluster spreadacross three VMs.

Next, “always invoked” may require security-relevant components(especially filters) are always executed, and best implemented in apipeline design pattern with a linear flow. Here, filters may refer todistributed firewalls or encryption routing technology that filterand/or block data communication. Filters may be implemented at everysingle workload within the disclosed computing platform system. Alwaysinvoked may be implemented with two or more components (usually filters)in parallel, which may make achieving a non-bypassable design morecomplex.

In some embodiments, certain components of computing platform 100 mayprovide domain separation within the integrated platform configuration.For example, filters may not be bypassed and are required for hosting ofa specific service. The software defined networking platform may beresponsible for facilitating the network traffic as well as implementingsecurity controls. If the specific service fails, the component may failsafe, disallowing network communication.

As a second layer of system protection, security hardening andconfiguration ensures that all aspects of computing platform 100 aresecure and utilize defense-in-depth throughout their implementation.

As the third layer of system protection, dedicated encrypted tunnels maybe implemented in computing platform 100 to represent all of softwaredefined data center (SDDC) components that provide security relevantservices to the access portion of the SDDC. This is a superset offeatures, tools, and settings that provide the network separation andisolation for the virtualized networks. One of the functions ofdedicated encrypted tunnels may include maintaining the virtualizednetworks communication and contents securely separated from each other.Each portion of the dedicated encrypted tunnel may be configured toprovide an interlocking piece of the platform security configuration.Failure of a single component does not compromise the integrity of othercomponents.

Configuration of the components may be managed across separatesubsystems with differing authentication and management mechanismsincluding: hypervisor, SDN, physical network management and dataswitches, management, RBAC rights management tools, and activedirectory. In one embodiment, a dedicated encrypted tunnel may bepreconfigured to work with each of the systems potential network ordomain (computing systems 1-n in FIG. 1 ). A network administrator maymanage visible network VM instances associated with each specificnetwork but cannot configure or modify the platform dedicated encryptedtunnel.

In one embodiment, dedicated encrypted tunnels may be configured toremain in a static configuration regardless of the virtualized networksutilizing the dedicated encrypted tunnel. Dedicated encrypted tunnelsmay contain no virtualized networks-specific data configurations. Thededicated encrypted tunnels acts as a neutral boundary, preventing datain a domain from traversing outside of the domain’s logical boundary.

FIG. 3 shows a high-level dedicated encrypted tunnel that may beestablished, e.g., via application servers) 104 and edge server(s) 106of FIG. 1 , in accordance with aspects of the present application. Inone embodiment, an end to end process for data traversing the disclosedsystem may involve the following:

1. Data may be transmitted through an external network switch 301 thatmay be physically connected to the computing platform 100 at one or morephysical network interface cards (NIC) 302 which may be processedthrough at least one virtual switch 303 that corresponds to its physicalNIC.

2. The virtual switch 303 may be configured to transmit data to anetwork specific service router 304 which may be the single logicaloutermost edge of the datacenter. Router 304 may be configured toprovide security controls to block unapproved traffic in accordance withpre-provisioned computing platform security policies. Once approved,data may be passed to a network specific logical switch 305 to connectwith the service router and SDN capabilities.

3. Virtualized encryption may then be applied to data 306 and networkspecific Tier 1 router 307 which may be configured to provideencapsulation and routing across cables 308 to the physical data switch309.

4. Data switch 309 may be configured to enable physical connectivitybetween the edge server(s) 106 in FIG. 1 and pass data traffic to theapplication server(s) 104 via dedicated data switch NIC 312 and thenetwork specific encryption routing 310 implemented on the applicationserver(s) 104,

5. Network specific encryption routing 310 and network specific router311 may be configured to pass only approved authorized data to thenetwork specific logical switch for access to network VMs 313, accordingto an embodiment of the present disclosure.

6. All data traversing the computing platform 100 may be encrypted usingencryption modules 314 from the VMs 313 and across physicalinfrastructure cables 315 into and out of the physical data switch 309.

7. Data into and out of the computing platform system 100 will alwaysoccur at the edge NIC 302 and into an external physical switch 301 fortransmission to external destination address.

The following table describes the components of the dedicated encryptedtunnel in FIG. , 3 . Here, East-West traffic may denote a direction oftraffic flow between VMs within the disclosed computing platform. Basedon a selected system topology deployed within a data center, East. Westtraffic may indicates the flow of data between the application servers(e.g., application server(s) 104 of FIG. 1 ). As a result ofvirtualization within the disclosed computing platform andhyper-converged infrastructure, East/West traffic may indicate thatenclave virtual workloads attached to virtual networks that communicatestrictly within the scope of the application server(s) 104, In oneembodiment, East/West traffic may include all SDN components such asvirtual firewalls, load balancers and enclave encryption methods.

Further, when data must enter or exit the disclosed computing platform,the flow of data may be defined as North South traffic, which mayindicate a data flow that either enters or leaves a corresponding datacenter from or to a system physically residing outside the computingsystem. In one embodiment, all traffic flowing North/South may traversefrom the origination or destination of a server (e.g., applicationserver(s) 104 of FIG. 1 ) through another server (e.g., edge server(s)106 of FIG. 1 ) then arriving on the physical network device external tothe system’s boundary.

# Device Description 301 Virtualized network switch (network specific)Represents the virtualized networks LAN portion of the physical networkinfrastructure, 302 Network-specific edge NIC Connected to the networkswitch, these NICs (two per virtualized networks for resiliency) arededicated to a single network. A virtual NIC (vNIC) is connected to thephysical NIC (hypervisor kernel). 303 Network-specific virtual standardswitch Provides virtual connectivity between the virtualized networks.304 Network-specific service router Acts as the single logical outermostedge of the data center. Provides a dedicated network-specific softwarefirewall that restricts traffic in accordance with approved ports,protocols, and services. Additionally, a dedicated edge VM firewallwhitelists approved ports and protocols and blocks all other traffic.305 Network-specific logical switch Provides connectivity between theservice router and SDN-based routing on the edge server(s) 106 306Network-specific encryption In conjunction with the Tier 1 router,provides routing on edge server 106 and East /West virtualizedencryption 307 Network-specific Tier 1 router Provides North/Southencapsulation and routing on the edge server 106 308 Physical cablingThe Hypervisor infrastructure is connected via a protected cableinfrastructure secured inside the data center rack. 309 Physical switchProvides connectivity between servers 102 and 104 that part of computingplatform 100. The ports that are used to provide East/West connectivityare isolated from other traffic traversing the switch. 310 Network-specific Encryption Provides routing on application servers 104and virtualized network encryption. 311 Network-specific Tier 1 routerProvides North/South encapsulation and routing on application servers104 312 Network-specific virtual standard switch Connected via theinfrastructure to application servers 104 313 Network-specific virtualmachine Built and managed to platform architecture standard 314Network-specific encrypted virtual traffic All East/West and North/Southtraffic that traverses the infrastructure switch is encrypted usingcommercial-off-the-shelf based encryption modules 315 Transport Zone Theencrypted tunnel for all enclave data connections from the physicalinfrastructure data switch to the edge host

FIG. 4 shows an example VM East/West network traffic, according toaspects of the present disclosure. The diagram shows the East/Westdedicated encrypted tunnel protection for traffic going between twoapplication servers 104 disclosed in connection with FIG. 1 . Thedisclosed computing platform 100 may be configured to provide multiplelayers of protection to prevent network interaction between virtualizednetworks that are simultaneously operating. The following describes eachlayer:

Layer 1 401: SDN virtualized network encapsulation may provide thefoundational separation of the virtualized networks traffic. A separatelogical network may be created for each virtualized network. Thisprocess ensures that only VMs connected to same logical network may bepermitted to communicate with each other. Specific headers and footersof each data packet in the network traffic may be inserted at Layer 1(no redirection required) in a communication path between the vNIC and acorresponding VM to ensure that all traffic is fully inspected by afirewall before any traffic may be permitted to proceed to the vNIC. Inother words, the disclosed computing platform 100 may increase the sizeof each data packet to allow for SDN.

Layer 2 402: A network encryption component may be configured to protectand encrypt the East/West network traffic. For example, VMs may onlycommunicate with VMs on the same virtualized network. As will bedescribed fully below, rules may be implemented to explicitly allowencrypted VM-to-VM traffic on the same virtualized network, and preventall undefined traffic using a deny all rule.

Layer 3 403: Dedicated encrypted tunnels-specific protection may beconfigured to ensure that VM traffic may only flow between VMs in thesame virtualized network including but not limited to providing rulesbased on VM-specific properties (e.g., name, security group, or OS) inaddition to properties like Internet Protocol (IP) address or portnumber used in legacy firewalls. Example rules may include explicitlyallowing encrypted VM-to-VM traffic on the same virtualized network, butpreventing all undefined traffic using a deny-all rule.

Layer 4 404: Applications, services, and ports may be identified andallowed to communicate only with authorized systems.

Layer 5 405: Application or service traffic flows between virtualizednetworks 1 and 2.

FIG. 5 shows additional physical protections for the East/West trafficacross an infrastructure switch which may be configured to isolate thetraffic from other groups of ports and provide interconnectivity betweeninternal components of computing platform 100.

A failure in any one of the five layers discussed in FIG. 4 may causethat layer to fail safe and prevent traffic from moving on the effectedVMs or networks. Such layer failure may not have an adverse effect onthe operation of the other four layers of security. However, traffic maynot flow if it cannot traverse every layer completely. In oneembodiment, SDN routing and software firewall may each drop a packet atthe layer where a failure occurs, so network traffic cannot passthrough. For example, the application server(s) 104 of FIG. 1 may beresponsible for implementation of policy security controls and may beconfigured to report a monitored failure to the management server(s) 102to disallow cross domain traffic.

Outside of virtualized networks-specific physical switches, computingplatform 100 may be enclosed in one standard rack. In one embodiment,tamper tape may be placed over the bezel of the platform. Anadministrator may visibly see if a bezel has been removed from any ofserver 102, 104, or 106, exposing storage and other critical componentsto unauthorized access, A bezel is most simply defined as the outsideframe around an object or device. In the case of data centers, a serverbezel is a front cover or faceplate found on the front of a server.Further, easily identifiable labels for each piece of network cablingreduce the chance of human error that may result in a misconfiguration.USB ports on computing platform servers 102, 104, 106 have been disabledvia, e.g., a remote access controller system. Administrators do not havepermission to enable these ports. If someone attempts to re-enable ormodify the USB ports, an error message may be generated, displayed andlogged to the remote access controller system lifecycle controller log.

In accordance with aspects of the present disclosure, the integrity ofthe physical system of computing platform 100 may be protected by thefollowing restrictions and conditions.

First, a secure booting system integrated with a unified extensiblefirmware interface (UEFI) may be configured to protect variousmanagement server(s) 102, application server(s) 104, and edge server106, If any system alteration is detected at startup, the system “purplescreens” and crashes. A change to the core virtualization software 102a, 104 a, 106 a operating system requires a reboot. Therefore, thesystem is protected against real-time changes. The secure boot functionof the virtualization software 102 a, 104 a, 106 a may perform anintegrity check of the core hypervisor operating system, other drivers,and system files during boot-up. RBAC rights management tools mayroutinely scan the hypervisor configuration (e.g., virtualizationsoftware 102 a, 104 a, 106 a in FIG. 1 ) to monitor and detect real-timealterations or configuration drifts in these hypervisor. Both subsystemsreport issues or configuration concerns to the audit tracking system.Further, TPM/TXT and RBAC management tools integration may ensure thehypervisor hardware platform has not changed since the last boot andprovide alerts and notifications in the event of system changes. Theyalso ensure that the configuration is compliant with cybersecurity bestpractices settings and prevent drift. In addition, in some embodiments,computing platform 100 may be configured to monitor the integrity of thefile systems for the following system components (e.g., via advancedintrusion detection environment (AIDE) software or any other suitablesoftware): SDN Manager, RBAC rights management tools, SDN Controllers,and SDN key manager.

Referring back to FIG. 1 , in one embodiment, computing platform 100 mayinclude two management servers 102, two application servers 104, an edgeserver 106, two physical network switches (not shown). All of theseservers may include Intel Boot Guard, which extends the platform root oftrust to the platform controller hub (PCH). In one aspect, the PCH maybe configured to contain one-time-programmable fuses that are burned,along with the boot guard policy and the hash of the master public key,at the OEM hardware factory during the EMC manufacturing process. Thekey manifest on the BIOS SPI flash is signed by the master OEM key, anddelegates authority to the boot policy manifest key. Then the bootpolicy manifest authorizes the initial boot block (IBB), which is thefirst BIOS code module to execute at reset vector. If the IBB failsauthentication, boot guard may be configured to stop the hypervisors(e.g., virtualization software 102 a, 104 a, 106 a in FIG. 1 ) and itscorresponding servers from booting. Each BIOS module may be configuredto contain a hash value of the next module in the chain and uses thehash value to validate the next module. The IBB validates (SEC+PEI)before handing off control to it. The (SEC+PEI) then validates(PEI+MRC), and (PEI-MRC) further validates the (DXE+BDS) modules. Ifenabled, the UEFI Secure Boot then extends the root of trust to theremaining BIOS, third-party UEFI drivers, and OS loader.

In one embodiment, each of the two management servers 102 may beconfigured to run virtualization software 102 a. and function to housethe systems and applications needed to administer the rules and securitymechanisms for computing platform 100. These out-of-band managementservers 102 may be accessed remotely or from any virtualized networks1-n. Out-of-band management may generally refer to a solution thatprovides a secure dedicated alternate access method into the disclosedcomputing platform management infrastructure to administer connecteddevices and management plane assets without having direct access toanything related to network and user data. For systems management of thedisclosed computing platform, out-of-band management may involve the useof various management interfaces dedicated to a management plane of thecomputing platform for an administrator to leverage in order to makechanges it updates to the system. Among other things, out-of-bandmanagement may allow a network operator to establish trust boundaries inaccessing various management related interfaces on the system. Further,the management plane of the present application generally provides asingle application programming interface (API) entry point to thesystem, persists user configuration, handles user queries, and performsoperational tasks on all management, control, and data plane nodes inthe system. Dedicated physical and virtual networking infrastructuremade available only to the administration of the platform.Administrators of the disclosed computing platform interface with allmanagement tools via this medium. No direct data path may exist betweenmanagement networks and virtual networks provided to the hostedenclaves.

Application servers 104 in a cluster may be configured to runvirtualization software 104 a. This cluster may be used to provision thenetwork, compute, and provide storage associated with each active andstandby virtualized networks 1-n.

Edge server(s) 106 may be configured to provide the only physicalexternal connectivity for computing platform 100. Server 106 may be thevirtual to physical demarcation point for virtualized networks trafficas it traverses outside computing platform 100 approval boundary and thevirtualized networks LAN (including physical switches, workstations, andWAN transport connections). Dedicated NICs (e.g., two NICs) pervirtualized network may be implemented to ensure physical separation andresiliency as network traffic traverses outside the network’s boundary.

Referring to FIG. 6 , a representative model of an edge server’sexternal connections and external networks is illustrated, according toaspects of the present disclosure. In one embodiment, edge server 106may be configured with a selected number of NICs 601 (e.g., 14 NICs) toprovide redundant networking support for up to four active networks.NICs 601 may be the only physical external connections to the systemreside at the edge server 106, which enforces the computing platform 100pre-provisioned policies to establish network traffic flow. 602indicates a definitive boundary to the platform 100 policy controls. Allresources existing beyond this demark point 602 may be controlled andconfigured by that network enclave’s administrators. 603 may illustratethe physical infrastructure dedicated to the enclave users, which mayinclude network transport capabilities. Therefore, 603 and beyond may beagnostic hardware in a network environment and does not adversely affectthe security controls of the security boundary of the computing platform100.

In one embodiment, computing platform 100 may include two physicalnetwork switches, one for management and storage traffic and the otherstrictly for network data traffic. Using two separate switches mayensure that network data traffic is isolated from management and storagetraffic. Both switches may be configured to have enhanced security inaccordance with relevant regulation, guidelines, and security bestpractices. In one embodiment, the following virtual local area networks(VLANs) may reside on the physical network management switch: hypervisormanagement, OBM VM, OBM host vMotion, application host vMotion,management, storage area network (SAN) 0, storage area network (SAN) 1.The VLAN for network data may reside can the physical network dataswitch. These switches may be divided into smaller zones of trafficusing ACLs and port security.

Administrative access to the data and management physical switches maybe restricted to a computing device (e.g., a laptop), and it is furtherprotected by requiring a user name and password. Password settingsconform to strict standards with a minimum length of 15 characters,which must include: at least two uppercase and two lowercase characters,two numbers, and two special characters. The physical switches may beconfigured to support administrative access via the front serial consoleport, a rear OOB Ethernet port, and an in-band connection. Both the rearport and in-band connection may be disabled to restrict access to theserial console port.

Port security may refer to the ability to limit the mandatory accesscontrol (MAC) addresses allowed to source packets on a physicalinterface. The physical switches may be configured to obtain necessaryMAC addresses and record them to a configuration file, such that onlythe recorded addresses may pass traffic on specific switch interfaces.If an unknown MAC appears on an interface, the physical switches may beconfigured to shut down the interface and generate a message detailingthe violation. An administrator must physically access the physicalswitches, log onto the console port, and specifically enable the downedport for traffic to pass again.

In one embodiment, ACLs may be configured to operate at Layer 3 shown inFIG. 5 and only allow for traffic that facilitates host traffic withinthe system. Violations of these ACLs may be logged in the form of Syslogmessages and forwarded to auditing and tracking tools.

The physical switches may have at least one USB port on a rear end ofthe unit that provides a simple method of moving configuration files toand from the switch. The USB port may not provide access to the switch.The USB must be mounted before a connected device can be used, and thisrequires administrative access to the switch. The USB port may bedisabled as part of the baseline configuration and can only be activatedby an authorized administrator who is locally logged onto the switch.

In accordance with aspects of the present disclosure, computing platform100 may include a number of software subsystems: hypervisor, managementserver appliance. RBAC rights management tools, SDN, AD, auditing andtracking tools, HCI Software, integrated remote access controller(iDRAC), infrastructure switch, and virtualized networks VMs. Each ofthese software subsystems will be described fully below.

A type 1 hypervisor may include and integrate vital OS components, suchas a kernel. By consolidating multiple servers onto fewer physicaldevices, hypervisor reduces SWaP and IT administrative requirementswhile driving high-speed performance. Computing platform 100 may use oneor more hypervisors, such that multiple virtualized networks 1-n may beexecuted on the same hardware. The hardware platform of computingplatform 100 may be virtualized and network VMs may be maintainedseparately while achieving performance and security requirements. Thatis, a hypervisor may be configured to create and run VMs and virtualappliances. The hypervisor may use virtualization to transformindividual virtualized networks into aggregated computinginfrastructures that include CPU, storage, and networking resources.Hypervisor may manage these infrastructures as a unified operatingenvironment. The hypervisor provides the virtual security architecturefor various hypervisor servers (management, application, and edgeservers 102, 104, 106 of FIG. 1 ), including but not limited to: VMisolation, native network isolation of the hypervisor kernel, virtualnetworking layer, management interface firewall, lockdown mode, VMencryption, and secure boot. VM isolation is described in the followingtable:

Element Description Virtualization extensions These technologiesautomatically trap sensitive events and instructions, eliminating thesoftware overhead of monitoring all supervisory-level code for sensitiveinstructions. In this way, VT-x and AMD-V give the Virtual MachineManager (VMM) the option of using either hardware-assistedvirtualization or binary translation, depending on the workload.Instruction isolation Intel VT-x and AMD-V extensions don’t enable VMsto run at protection Ring 0. Only the VMM runs at a hardware privilegelevel; guest operating systems run at a virtualized privilege level.Memory isolation The Hypervisor software allocates memory when itdefines the resources to be used by the VM. A guest OS uses physicalmemory allocated to it by the Hypervisor software and defined in theVM’s configuration file. Memory Protection To protect privilegedcomponents, such as VMM and hypervisor software Hypervisor software, thehypervisor uses address space layout randomization. (ASLR). Itrandomizes where core kernel modules are loaded into memory. The NX/XDCPU features enable the hypervisor software to mark writeable areas ofmemory as non-executable. Both methods protect the system from bufferoverflow attacks in the running code. Device Isolation Each VM isisolated from other VMs running on the same hardware. VM share physicalresources such as CPU, memory, and I/I devices; but a guest OS in anindividual VM cannot detect any device other than the virtual devicesmade available to it.

Just as a physical machine can communicate with other machines in anetwork only through a network adapter, a VM may communicate with otherVMs running on the same or a different hypervisor host only through avirtual switch. Further, a VM communicates with the physical network,including VMs on other hypervisor hosts, only through a physical networkadapter. The virtual networking layer may include virtual networkdevices through which VMs interface with the rest of the network. Thehypervisor may rely on the virtual networking layer to supportcommunication between VMs and their users. In regards to VM isolation ina network context, the following rules may be applied; i) If a VM doesnot share a virtual switch with any other VM, it may be completelyisolated from other virtual networks within a server; ii) If no physicalnetwork adapter is configured for a VM, the VM may be completelyisolated from any physical or logical networks.

Using a vNIC level firewall, a VM may be isolated from other VMs, evenon the same switch (Layer 2 isolation). These rules may be applied tothe vNIC of the VM, not at the switch, enabling them to travel with theVM.

Further, the virtual networking layer may include various virtualnetwork devices used by VMs to interface with the rest of the network.The hypervisor relies on this layer for communication between VMs andtheir users, and the hypervisor hosts use it to communicate with iSCSIstorage area networks (SANs) and network-attached storage (NAS).

Computing platform 100 may use virtual switches in the following areas:hypervisor distributed switch for VMs on management servers and for thehypervisor management; SDN hypervisor distributed switch on theapplication servers; and standard virtual switch or distributed virtualswitch on edge server(s) 106.

Each virtual switch may be configured to provide local networkconnectivity for the local VMs on the local host(s). For instance, themanagement VMs remain isolated to local traffic on the backendinfrastructure.

According to aspects of the present disclosure, a management interfacefirewall built into the hypervisor may be configured to restrict networkaddresses that can connect to the management interface. Withcommunication to the hypervisor host over the management interfacerestricted by transmission control protocol (TCP) port number,functionality other than administration is enabled, including hypervisorvMotion and high availability, hypervisor fault tolerance, IP-basedstorage, and other basic functions such as domain name system (DNS),logging, and network time protocol (NTP). This management interfacefirewall may be enabled and configured on computing platform 100 toprovide defense-in-depth on the management plane.

Moreover, a lockdown mode is a feature of hypervisor that disableslog-on and API functions from being executed directly on a hypervisorserver. It is available only on hypervisor servers that have been addedto management server 102. When a server is in the lockdown mode, userscannot run hypervisor command line interface (CLI) commands from anadministration server or from a script. Use of lockdown mode may removeall hypervisor API privileges associated with root. In a lockdown mode,only the service account has authentication permissions. No other userscan perform operations against the server directly. The root accountcannot be logged onto and API operations cannot be run using it, makingit a key component of computing platform 100 isolation.

The hypervisor VM encryption component may be configured to ensure datais kept secure. When computing platform 100′s VMs read/write to theplatform storage system, data may be encrypted in motion (across thestorage backplane) and at rest (on the disks). Data coming from a VM isencrypted before it stored in the VM disk (VMDK).

In one embodiment of the present disclosure, if VMDK data is accessed byany unauthorized entity, only meaningless data may be shown. The VM thatlegitimately owns the VMDK has the necessary key to decrypt the datawhenever that data is read and then fed to the guest OS usingindustry-standard encryption algorithms to secure this traffic withminimal overhead. In one embodiment, computing platform 100 may useprocessors that support the Intel Advanced Encryption Standard - NewInstructions (AES-NI) set to speed up the encryption/decryptionoperations. Only privileged users that are assigned the cryptographicoperations can perform those functions. The privilege set may befine-grained, and the default platform administrator role may includethese privileges. The community of interest (COI) administrator rolecannot change cryptographic operations.

FIG. 7 shows an overview of VM encryption, in accordance with aspects ofthe present disclosure. In one embodiment, VM encryption components mayinclude an external KMS 701, management server 102, and hypervisorservers. The management server 102 may be configured to requests keyfrom the external KMS 701, which may be configured to generate and storethe keys and pass them to management server 102 for distribution. Themanaged VM maintains all KMS keys 702 for distribution to any KMSclient. Virtual servers may be encrypted with these keys, and for moresecurity, the encryption may utilize not only the encryption key, butalso a block’s address. This means two identical data blocks of a VMdisk 703 with the same content may result in different encryptionvalues. To prevent a compromise of the encryption keys and the encrypteddata, each component may be hosted from different hypervisor servers.All KMS systems and policies maybe a preconfigured feature of thecomputing platform, and the keys and encrypted VM may be on separatevirtual management servers by pre-provisioned computing platform 100policy. All encryption policy application happens via trusted connection704 over KMP for key management.

VM encryption may support the encryption of VM files, virtual diskfiles, and core dump files. Log files, VM configuration files, andvirtual disk descriptor files may not be encrypted because they containnon-sensitive data and may be used in disk management. VM encryptionuses hypervisor APIs for I/O filtering, which allows VM data to beintercepted in the virtual small computer systems interface (vSCSI)emulation layer.

The vSCSI layer may include a layer in hypervisor that resides betweenthe VM and the VM file system (VMFS). The I/O filter framework, which isused to implement services like encryption, caching, and replication,may be implemented entirely in user space. This may allow the VM data tobe isolated from the core architecture of hypervisor, eliminatingpotential issues to the core functionality of the hypervisor. In theevent of a failure, only the VM in question may be affected.

In accordance with aspects of the present disclosure, applicationservers and data hosts may access their underlying storage systems, asshown in FIG. 8 , and there may be multiple filters enabled for a VM ora VMDK, and these filters may be sequentially chained 801. Data may beprocessed serially by each of these filters, then passed to VMFS if notstopped by a filter. Industry standard protocols 802 may be leveraged toaccess storage and the VM may have its I/O safely and securely filteredin accordance with a policy when applied, such that data may beexchanged between IOFilter and vSphere APIs for I/O filtering (VAIO).HCI virtual controllers may gain secure access to storage devices to bemade available as a collection of disks spread across multiple storagenodes. Each VM may be provided direct communication channel 803 tounderlying storage subsystem. These protocols are leveraged specificallyby the hypervisor to ensure all storage related data may be subject topolicies enforced to the hypervisor via the IOFilter and includes VMspecific disk storage processes.

FIG. 9 shows a simplified topology of the KMS. In one embodiment, KMS901 may be a secure, centralized repository of cryptographic keys forcomputing platform 100. Due to redundancy, computing platform 100 mayhave more than one KMS configured with a management. One of the KMS 901may be designated as the default in management. For example, KMIP v1. 1compliant KMSs may be supported, and management server 102 of FIG. 1 maybe the client of KMS. KMIP may enable management server 102 tocommunicate with any KMIP-compliant KMS vendor as part of virtualmanagement 902. Trusted network connection 903 may be configured tofacilitate the connectivity between the KMIP services running on theHypervisor 904 and the KMS Server Virtual Management 902.

Two types of keys may be used for VM encryption:

-   i) Data Encryption Key (DEK): the application server 104 and its    hypervisor may be configured to generate and use internal keys to    encrypt VMs and disks; and-   ii) Key Encryption Key (KEK): The management server instance    requests AES-256 keys from the KMS. The management server 102 may    store the ID of each KEK, but not the key itself.

Initially, the hypervisor hosts may not have the necessary keys toperform cryptographic operations like encrypting and decrypting guestdata. The management server 102 may obtain the keys from the KMS andforward them to the hosts using the KEKs. The host generates the DEKs,which are then used for encrypting and decrypting VM files,

KEKs may be used to encrypt the DEKs, and these encrypted DEKs arestored in configuration files. Once encrypted, the KEK for the VM needsto be in Hypervisor memory for the VM to be powered on. If for somereason the Hypervisor host is power cycled or the encrypted VM isunregistered and then re-registered, management server 102 may beconfigured to obtain the KEK from KMS again and push it to Hypervisor.KEKs may be stored only in the KMS where they are generated and notpersisted anywhere on hypervisor. KMS should be highly available, orkeys should be replicated between multiple KMS instances added to thesame KMS cluster for accessibility of KEKs.

During the encryption process, different hypervisor components interactas follows:

a). When the maintenance administrator performs an encryption task, suchas creating an encrypted VM to support a new virtualized network oncomputing platform 100, management server 102 may be configured torequest a new key from the default KMS. This key is used as the KEK.

b). Management server 102 may be configured to store the key ID and passthe key to the Hypervisor host. If the Hypervisor host is part of acluster, management server 102 may be configured to send the KEK to eachhost in the cluster. The key itself is not stored on the managementserver system 102. Only the key ID is known.

c). The hypervisor host generates DEKs for the VM and its disks. Itkeeps the internal keys in memory only and uses the KEKs to encryptinternal keys. Unencrypted internal keys are never stored on disk. Onlyencrypted data is stored. Because the KEKs come from the KMS, the hostcontinues to use the same KEKs.

d). The hypervisor host encrypts the VM with the encrypted internal key.Any hosts that have the KEK and can access the encrypted key file canperform operations on the encrypted VM or disk.

In one embodiment, computing platform 100 may be configured to usehypervisor U-EFI secure boot to verify the integrity of the hypervisorhosts at startup.

The Hypervisor servers may include the following components, each ofwhich is cryptographically signed:

UEFI Component Description Boot Loader The Hypervisor boot loader issigned with the Microsoft UEFI Public Certificate Authority (CA). Thisensures that standard UEFI secure boot firmware can validate the bootloader. The boot loader code also contains a public key, which is usedto validate the VM kernel. Secure boot verifier The secure boot verifiervalidates every cryptographically via at least one public key.Installation Bundle (VIB) A VIB is a file archive (TAR g-zipped file),an XML descriptor file, and a digital signature file. When Hypervisorboots, it creates a file system in memory that maps to the contents ofthe VIBs. If the file never leaves the cryptographically signed archive,then you don’t have to sign every file, just the archive. The VIBs aresigned with the public key and validated with the secure boot verifier.

An example secure boot process may be the following: host powers on,UEFI validates the hypervisor boot loader against a digital certificate,hypervisor boot loader validates the kernel against the digitalcertificate, kernel runs the secure boot verifier, secure boot verifiervalidates each VIB against the digital certificate, hypervisor serverboots up. FIG. 10 shows a high-level overview of the UEFI secure bootVIB certification process.

Reference Process Step Process Description 1001 Running hosted /DCUI/VM’s Secure Boot Verifier validates all VIB’s against digitalcertificate 1002 Secure Boot Verifier VM Kernel starts Secure BootVerifier which also contains digital cert 1003 VM Kernel Boot Loadervalidates VM Kernel against digital certificate 1004 Boot Loader BootLoader contains digital certificate 1005 UEFI Firmware UEFI firmwarevalidates Boot Loader against that digital certificate 1006 HardwareVendor supplied UEFI firmware contains digital certificate

2. The virtual management server may include a pre-configured Linux VM,which is optimized for running management server 102 and the associatedservices on Linux. The virtual management server may be configured toenable the management of Hypervisor hosts. Each hypervisor server may beset in a lockdown mode, forcing administration through the virtualmanagement server console. RBAC management tools may provide anadministrative proxy to the administrative processes.

On computing platform 100, virtual management server may be used tomanage a network’s multiple hosts and pool resources. Installed on amanagement server 102, virtual management server may centralize theoperations, resource provisioning, and performance evaluation of VMsresiding on computing platform 100. It provides a central managementconsole to manage all the system’s VMs.

In accordance with aspects of the present disclosure, the virtualmanagement server may be configured to provide statistical informationabout the resource use of each VM and adjusts compute, memory, storage,and other resources from a central application. It may manage theperformance of each VM against specified benchmarks, and optimizeresources wherever required to provide consistent efficiency throughoutthe networked virtual architecture. Besides routine management, thisvirtual center may be configured to ensure security by defining andmonitoring access control (AC) to and from the VMs, migration of livemachines, and interoperability and integration among other Web servicesand virtual environments.

On computing platform 100, virtual management server administration maybe controlled using defense-in-depth with RBAC management tools as aproxied instance.

3. RBAC management tools is a virtual appliance that resides betweenadministrators and the hypervisor to add critical role-based accesscontrols, visibility, and secure multi-tenancy to the virtualinfrastructure. RBAC management tools is a proxy, separating operatorduties and monitoring access to ensure that only authorized personnelconfigure security controls and policies. Users are unaware that theyare going through this proxy.

On computing platform 100, RBAC management tools may be configured tolimit and control the functions of the hypervisor environment. This mayenable a second level of control for objects within the hypervisorplatform and prevent accidental or intentional attempts atmisconfiguration of the platform. Furthermore, RBAC management toolsroles and permissions limit users to routine operational functions. Withits expanded logging capability, it records and tracks potentialmisconfiguration attempts.

In one embodiment, RBAC management tools may be configured to determinespecifically what a user is allowed to do after logging onto thedisclosed computing platform. RBAC management tools may be configured toauthenticate, control, and apply additional RBAC controls to hypervisorfunctions during normal operation of the platform, RBAC management toolsmay also be configured to provide additional safeguards to preventaccidental or malicious configuration changes by the administrator. Inone aspect, RBAC management tools may include a virtual appliancedeployed as a transparent proxy that allows for a single-entry point andnon-intrusive application of security controls to all administratoractions within the hypervisor environment. Fundamentally, any actionthat is issued by an administrator is proxied, evaluated, logged, andforwarded to the virtual management server (if approved).

RBAC management tools may be implemented within the hypervisormanagement network and only interacts with administrator actions, sothere is no impact on VM performance or network traffic. The RBACmanagement tools solution may provide many capabilities, includinggranular RBAC, object-based access controls, secondary approval, auditquality logging, hypervisor configuration hardening, and trustattestation service. These capabilities may allow computing platform 100to secure the contents of each virtualized network 1-n at thevirtualization layer. An example RBAC management tools architecture isillustrated in FIG. 11 . Referring to 1101, the specific network enclaveadministrator may assign pre-defined RBAC roles available to thecomputing platform 100 that are accessed via web or direct client 1102.The RBAC management policies may be pre-defined and automaticallyapplied to each capability of computing platform 100, the management;application; and edge 1103 components. Security controls may bepre-provisioned by policy to ensure no direct access 1104 outside ofenclave network administrator RBAC capabilities are authorized whichmaintains the integrity of the RBAC and enforces the pre-approvedpolicies 1105 for logging and authentication that are specific andindependent to each individual network on the computing platform 100.

In one aspect, RBAC management tools transparent proxy may act as acentral point of control by intercepting, monitoring, enforcing, andlogging all administrator requests originating from the hypervisor WebClient and the hypervisor HTML5 Web Client (hypervisor Client and SSHare disabled on this system). The RBAC management tools proxy appliancemay be deployed in active-passive pairs, so that if the primary (active)appliance fails, the secondary (passive) appliance takes over thenetwork identity and serves as the primary RBAC management toolsappliance.

Furthermore, RBAC management tools policy engine may add anotherenforcement mechanism that enables computing platform 100 to implementsecurity best practices such as “separation of duties” policies thatkeep privileged users in their swim lanes or the implementation of the“principal of least privilege” within a given role, thereby enforcingthe mechanism of the dedicated encrypted tunnel.

In one embodiment, the RBAC management tools policy engine may beconfigured to allow computing platform 100 to limit privileged useraccess to objects with or without a certain label (such as Alpha domainor Bravo domain). A Bravo network administrator, for example, cannotadminister, access, or even view Hypervisor resources from any otherdomain except Bravo. VMs and other resources are tagged. When theadministrator logs on, only objects tagged “BRAVO virtualized networksVM” may be visible.

Additionally, RBAC management tools may constrain a user to access only“BRAVO virtualized networks VM” objects when coming from the known IPrange/subnet. This functionality allows computing platform 100 tologically segment each virtualized networks VM by the logical boundariesthat define it. Access to the RBAC management tools console is limitedto a management laptop.

Next, RBAC management tools may be configured to enforce access rightsand security compliance to align with the system/security requirementsof each administrator role. The RBAC functionality may be integratedinto the system to provide granular control of security policies,restricting administrative rights to specific roles and work functions(or tasks),

In one aspect, the RBAC capabilities may be integrated with virtualresources through the virtual management server management system toinclude VMs, port groups, virtual switches and Hypervisor hosts. TheseRBAC capabilities may be configured within the management domain AD withthe appropriate users tied to one or more groups designated on thesystem to enforce policy and controls. The end result of this is thatthe dedicated encrypted tunnels will be able to keep virtualizednetworks user access separated and only permit validated admin roles toaccess the enclave that they are administering.

In one embodiment, RBAC management tools may include five elements:objects, groups, roles, labels, and rules. In one embodiment, theobjects may include the VMs and virtual networks; the groups may referto the collection of users that have access to the resources; the rolesmay include policy that restrict/allow access to specific areas of thesystem; the labels may include the tagging of the objects; and the rulesmay include the policy mechanisms enforced from the disclosed securityarchitecture.

Further, the RBAC management tools authentication engine may beconfigured to provide an extra layer of security, along with the usualuser name and password. RBAC management tools validates privileged useraccess requests made through the RBAC management tools transparent proxyusing AD to verify proper group membership. RBAC management tools alsoensures that users accessing the disclosed computing platform areproperly authenticated.

In another aspect, RBAC management tools compliance engine may beconfigured to assess the system against the Hypervisor and VM containerSTIGs to assess compliance on a recurring basis. RBAC management toolsmay provide “out of the box” compliance templates for common controlbaselines, and these templates may be customized with organizationalspecific controls. For example, RBAC management tools may verify thatSyslog events are sent to a security information and event managementplatform, ensure NTP settings are correct, check time out settings, andallow only VIBs with the proper acceptance level. If the system isnon-compliant with any of the STIG settings, the compliance engineautomatically reapplies the settings to bring the system intocompliance.

In one embodiment, a RBAC management tools dashboard may display thecurrent and trending levels of compliance with STIGs and the number ofhosts that are in or out of compliance. RBAC management tools alsoprovides scheduled reports that show compliance drift over time and adetailed log of all transactions performed against the virtualenvironment.

In another embodiment, RBAC management tools may authenticateadministrators through Windows AD. At each login, an administrator maybe authenticated against her AD credentials. RBAC management tools mayuse a Service Account to query AD for authentication. RBAC managementtools intercepts all requests destined for RBAC managementtools-protected hosts, such as Hypervisor and the virtual managementserver, and authenticates the user against the Directory Service.Authentication of the user (including session ID) lasts for the fullRBAC management tools session. Once a session is established,authorization to perform an operation (including verification ofdirectory group membership) can occur multiple times in a session. AfterRBAC management tools authenticates the administrator, it performs anauthorization check for each request based on the local policy data. Ifauthorized, RBAC management tools may be configured to forward therequest (using a special service account) to a target server.

The following may be the authentication and authorization process usinga hypervisor client which may include software on an end user device anadministrator would use to access the entire system or administer one ormore networks. One may not use the hypervisor client to access anyenclave data.

A). RBAC management tools obtains the administrator’s identity duringlog on.

B). RBAC management tools queries AD to authenticate the administratorand validate the password. RBAC Management Tools also obtains groupmembership information to authorize attempted operations, such as thefollowing:

a. Identify the requested operation (such as start a VM).

b. Identify the object admin is targeting for an operation (such as VM‘mref 449’).

c. Query the RBAC management tools policy database to identify the listof admin groups authorized to perform the requested operation on thespecified object and determine if the current admin is a member of anauthorized user group.

d. Log information about the operation, the admin, and the objectinvolved.

If the user is authorized, RBAC management tools may re-issue theoperation request and send it to the management server 102 or clusteredhypervisor host where the original logon request was routed. Otherwise,RBAC management tools returns an error message to the user. If RBACmanagement tools cannot authenticate a user, then authentication fails,and the admin is denied access to the specified target.

4. RBAC rights management tools may be configured to enable computingplatform 100 to manage all their encryption keys at scale, how oftenthey rotate them, and how they are shared securely. For example, KMS maybe configured to provide keys to encryption acting as an independentprovider.

RBAC rights management tools may be a scalable KMIP server that servesas the KMS for VM encryption. RBAC rights management tools may providekey management services by automating the lifecycle of encryption keysand providing key storage, distribution, rotation, and key revocation.

Delivered as an open virtualization archive (OVA) file, RBAC managementmay be installed and connected to management as a KMS and configured inan active/active cluster scheme to ensure key availability tomanagement. The methodology of the key process is as follows:

a). The Hypervisor host (managed by management server 102) generates anduses an internal key, called the data encryption key (DEK), to encryptVMs and disks.

b). The management server 102 then requests a key from RBAC rightsmanagement tools. This key, known as the key encryption key (KEK), isthen used to encrypt the DEK.

c). The encrypted DEK and KEK ID are stored in the disk image metadata.

d). When a VM boots up on an Hypervisor host, or when the HCI data storeis brought online, the Hypervisor host reads the KEK ID and requests theKEK corresponding to the KEK. ID from management.

e). Management server 102 issues a request to RBAC rights managementtools for the KEK corresponding to the KEK ID and delivers the KEK tothe Hypervisor host.

f). The Hypervisor host uses the KEK to decrypt the DEK, and then usesthe DEK for encryption and decryption.

5. Network virtualization platform, SDN, creates network services likerouting, software firewalls, and other services that are software-basedand implemented on the hardware platform. Computing platform 100 itselfmay be portable to other hardware platforms.

6. AD is a directory service that provides identity-related services. ADDomain Services (AD DS) may authenticate and authorize all users andcomputers in a Windows domain type network—assigning and enforcingsecurity policies for all computers. Moreover, it may allow managementand storage of information, provide authentication and authorizationmechanisms, and establish a framework to deploy other related services.

In computing platform 100, AD may be installed in the managementenvironment to enable RBAC. AD may provide an authentication mechanismfor the administrator when using the RBAC management tools andhypervisor subsystems, and auditing and tracking tools. For example,each Windows server may use a pre-configured secure instance of WindowsServer 2016 as the base OS. In one embodiment, the management domain ADinstance may be based on the secured Windows 2016 baseline. Atdeployment, it may be scanned and configured to be compliant with thelatest baseline STIGS, which do not have non-administrative users anddoes not authenticate any user-based traffic. T he primary function ofthe AD instance may include authenticating three subsystems: RBACmanagement tools and its extension into hypervisor, and administrativeaccess to the auditing and tracking tools system.

7. The HCI Software provides performance, scale, efficiency, andresiliency with policy-based intelligence and automation, as well asstability and security. By leveraging DirectPath (hypervisor PCIePassthrough), HCI software bypasses the hypervisor for anystorage-related input/output (I/O) activities.

In computing platform 100, the HCI software OS may be used to create HCIstorage systems across the platform application and management serversthat are pre-allocated between the management environment and each ofthe provisioned virtualized networks 1-n.

HCI software may allow the storage of each physical server in the systemto be grouped into a single server that can be used by different VMs.Using HCI Software, computing platform 100 may be equipped with the HCI,which includes virtualized computing (a hypervisor), virtualized SAN(software-defined storage) and virtualized networking (software-definednetworking).

In one aspect, HCI Software with HCI may combine a hypervisorvirtualization technology with a linear-scaling all-flash storage array.Non-volatile memory express (NVMe) data paths and storage tiers withpolicy-based quality of service (QoS) may enable the consolidation ofmultiple, mixed application workloads. Available in an all-flashconfiguration, HCI Software leverages NVMe PCIe flash RAM and solidstate drives to automatically enhance workload performance.

Moreover, HCI software nodes may be clustered and each node’s capacity,input/output operations per second (IOPS), bandwidth, and cache areaggregated and available to any VM in the cluster. HCI software may beconfigured to use inline erasure coding to write and protect data. Thiserasure coding may be efficient when writing data, and also scramblesdata for increased security. Data may be partitioned into small blocksand scattered across all the disks. Without the key and a multitude ofserver appliances, it may be virtually impossible to recompose the data.This is one way that the dedicated encrypted tunnels keep user datasecure.

In one embodiment, the HCI software hardware may be operated from a VMrunning a heavily customized, Linux-based controller called the HCIsoftware OS. HCI software may maintain direct access to the underlyingstorage, bypassing a virtualization layer, and provide a storage clusteravailable to VMs running on any HCI software system (node) within acluster. Each node may contain a hypervisor on which the VMs execute.HCI software may be used to present storage to the hypervisor as anInternet small computer systems interface (iSCSI) target, simplifyingthe administration of connecting storage to VMs and eliminating the needto use complicated logical unit numbers (LUNs) masking to accessstorage.

Further, bi-directional challenge handshake authentication protocol(CHAP) may be used to authenticate iSCSI connections to the HCI softwarevolumes, where the management and virtualized networks data storesreside. Host access groups may be created in HCI software and are tiedto iSCSI initiators of the Hypervisor hosts. For example, a separatehost access group may be created for management and each virtualizednetworks volume. Each host access group may have different abidirectional secret, which separates the enclaves and management domainand ensures that they may only access their assigned storage, as shownin FIG. 12 . The Hyper-converged Shared Storage - Logical SoftwareDefined Storage configuration 1201 may be configured to leverage thephysical disk drives installed on each hardware server system to operateas one large storage device that stores all VMs for the solution. Thestorage system 1204 may be partitioned into storage pools dedicated tomanagement and application contexts, respectively. Security policies maybe applied to ensure data separation is maintained 1205. Separatemanagement cluster 1202 may be a group of physical servers that arededicated to the Administration Management Plane of the system. Theseservers provide all the necessary CPU, memory, and networking componentsneeded to run all the security software. Separate application cluster1203 may be a group of physical servers that are dedicated to theEnclave logical networking resources of the system on the Data Plane.These servers may be configured to provide all the necessary CPU,memory, and networking components needed to run the various, logicallyseparated, enclave networks.

The HCI software OS may combine the storage that is local to each serverand then pool it together into a single highly available storage system.Throughout the HCI servers, the solid state drive storage (volumes) maybe created and securely delivered to hosts that run application serversassociated to a secure enclave. This allows additional networks to beadded by simply adding another HCI appliance. The required highlyavailable storage and compute resources may be added to the networkwhile maintaining the required security.

Resources may be modularized when being placed into logical units calledvirtual performance groups (vPGs). In one aspect, vPGs may allowspecific combinations of resources to be logically grouped to providedifferent tiers of service. The shared storage may be distributed acrossthe entire vPG, and each VM may utilize the performance and capacity ofthe entire infrastructure. VMs never have to move data just because a VMwas moved on the hypervisor. By leveraging the resources of the entirevPG, performance improves for the whole infrastructure, not just wherethe fastest disks or processors reside. In the multi-tenant environment,separate vPGs may be deployed for the management construct, then fromwhere the network preside, ensuring physical isolation of managementcloud applications from individual enclaves.

The system uses, e.g., a native hypervisor iSCSI interface or anysuitable interface mapped to dedicated redundant 10Gbps NICs as thebackbone for all storage traffic within the vPG. Storage area network(SAN) traffic may be physically isolated from all VM and managementtraffic.

Network equipment may be resilient, providing multiple I/O paths, andscales with the size of the vPG. Since resilient equipment operates inactive/active mode, maximum bandwidth may be provided for each VM’scommunication.

Network communications may be handled peer-to-peer across nodes, whichmeans there is no master node to bottleneck traffic. The HCI softwarestorage infrastructure load balances all storage read and write trafficbetween two non-connected NICs that may be called SAN0 and SAN1. Thismodel may ensure survivability of data flow in the event of a failure,as well as load balances the storage data between links to optimizeperformance.

8. iDRAC may be configured to alert administrators to server issues andperforms hardware server management. iDRAC may be used to initiallyconfigure computing platform 100. Once the platform is operational,iDRAC may be configured to generate log files that report on the healthof the system’s servers. These files are processed by a platformauditing system,

iDRAC may be a piece of hardware that resides on server motherboards. Itprovides a secondary monitoring system that allows the underlyinghardware to be independently monitored from a UEFI subsystem. The iDRACsystem also monitors and prevents hardware or BIOS changes, and allowsfor monitoring of hardware faults in the server platform.

The iDRAC system enables the administrator to monitor the system healthof the management, application, edge servers, 102, 104, 106 of FIG. 1 .One of the edge server ports contains the iDRAC capability may beexclusively connected to the infrastructure switch. iDRAC monitors thephysical system for issues and hardware changes and sends alerts to anauditing and tracking tools server.

With limited access to the console, the administrator may be authorizedto: view managed server health, view sensor information such astemperature, voltage, and intrusion, monitor CPU state, processorautomatic throttling, and predictive failure, perform power-relatedoperations and monitor power consumption, view log event data in theiDRAC Lifecycle Controller log, and monitor and report any hardware orsystem changes to the underlying subsystem.

9. The Physical network infrastructure management switch suppliesnetwork interconnectivity between the disclosed computing platformcomponents and systems. It has no connectivity to virtualized networks1-n or any external network 1-n.

10. A management laptop may be the sole entry point into the OOB portionof computing platform 100. The software installed on the managementlaptop may be used to install and set up computing platform 100, as wellas complete daily administration of the OBM and auditing functions.

11. The virtualized networks VMs are the applications and services thatreside within the virtualized data center. VMs are provisioned viasubsystems outside the platform approval boundary and managed accordingto the virtualized networks and TTP standards. Internal operations ofthe VMs are not visible to the platform’s management domain which mayinclude a management network that is virtually separate with its ownphysical switch to administer all of the networks on the disclosedcomputing platform.

In one aspect, virtualized networks are VMs hosted on the applicationcluster which may include two HCI software servers running a type 1hypervisor. This cluster may be used to provision the network, compute,and storage associated with each active and standby virtualizednetworks. Computing platform 100 may support more servers withadditional hardware, but the servers must be installed and configuredbefore deployment.

The users on the virtualized networks 1-n may have accounts which ishosted on a network VM. The virtualized network’s AD domain may beadministered by administrative accounts local to its AD domain. Eachdomain may be separate and have no connectivity to other networks.

For instance, the privileged role called network admin bravo may onlyneed to access resources for the Bravo virtualized networks VMs. VMs andother resources are tagged, and when the admin logs on, only theeobjects tagged “Bravo virtualized networks VMs” are visible.

Additionally, RBAC management tools may ensure this user only accessesBravo virtualized networks VM’s objects when coming from the known IPrange/subnet. This functionality allows computing platform 100 tologically segment each virtualized networks VM set by logicalboundaries.

The OBM servers cannot be accessed remotely or from any virtualizednetworks 1-n. The mission of the networks is to host services (such asAD, SharePoint, and Call Managers) for users to access.

In accordance with aspects of the present disclosure, SDN may be thefoundational backbone of computing platform 100 infrastructure,providing isolated network communication inside the platform and routingfunctionality to services outside the data center. SDN virtualizednetworks leverages logical switches and distributed routers segregatedby multiple controls to prevent communication between networks. Separatetransport zones prevent enclave interconnectivity, and cryptographicallysegregates each network’s network stack, preventing anycross-communication should a hypervisor zero-day vulnerability beexploited to defeat the logical separation.

Since all SDN virtualized network controls are bifurcated from themanagement interface, the attack vector is dramatically restricted to aseparate and independent access console. The result is a robustvirtualized network platform, with defense-in-depth controls, allowingmulti-tenancy with a high level of confidence across consolidatedcoalition virtualized networks.

Computing platform 100 may be configured to use SDN virtualized networkto accomplish the following goals for part of the defense in depthlayers of the system: provide network virtualization central to thevitality of the entire system, ensure that the traffic is separated andremains separated, provide a reliable and stable platform, providemultiple layers of protection mechanisms and seamless integrationbetween virtual and physical devices, keep the communication pathsassociated with each virtualized networks 1-n secure from each other andfrom the OBM tools, provide a distributed firewall capability for eachVM at the hypervisor level, implement data encryption to encryptEast/West traffic for each virtualized network 1-n.

Building a multi-tenant SDDC platform may pose unique securitychallenges. A network’s VMs need a network path securely isolated fromthe physical switches on the network to the virtual NIC of the VM. Oncomputing platform 100, information may flow from a virtual virtualizednetwork to the edge server switch and external networks. Thisinformation may move over standard TCP/IP connections. Computingplatform 100 may use SDN software to provide an order of magnitude tothe level of separation.

SDN is the network virtualization platform used for this SDDC’,delivering networking and security entirely in software, abstracted fromthe underlying physical infrastructure. SDN accomplishes this is byimplementing a SDN abstraction of the physical network that can becontrolled through software.

SDN technology is an approach to cloud computing that facilitatesnetwork management and improves network performance and monitoring. SDNaddresses the static architecture of traditional networks, which isdecentralized and complex, by improving flexibility and troubleshootingcapabilities.

In general network implementations, SDN may be required to allow forpredefined dedicated encrypted tunnels, and for the SDN process to buildand maintain separation. In general, SDN centralizes networkintelligence in one network component by disassociating the forwardingprocess of network packets (Data Plane) from the routing process(Control Plane). In accordance with aspects of the present disclosure,data plane may perform stateless forwarding and/or transformation ofdata packets based on tables populated by the control plane and reporttopology information to the control plane, and maintain packet levelstatistics. The data plane may be the source of truth for the physicaltopology and status for example, VIF location, tunnel status, and so on.For all data going to or coming from enclave resources this data isconsidered Data Plane traffic. The data plane may also maintain statusof and handles failover between multiple links and or tunnels. Controlplane may include a communication path available only to the source codeof the various products. The control plane may be split into two partsin SDN, a central control plane (CCP), which runs on the controllercluster nodes, and a local control plane (LCP), which runs on thetransport nodes, adjacent to the data plane it controls.

In one embodiment, the control plane may include one or more controllerswhich are the brain of SDN network (in other words, where the wholeintelligence is incorporated). This intelligence centralization has itsown requirements for security, scalability, and elasticity.

With SDN, computing platform 100 virtualizes the network that is used byits virtualized networks 1-n to communicate internally. SDN managervirtual appliance and controllers reside on management servers 102 forindependent management and execution of the SDN routing subsystem. Thissetup is pre-configured, and the administrator cannot make changes tothis subsystem. I

In much the same way that server virtualization programmaticallycreates, snapshots, deletes and restores software-based VMs, SDN networkvirtualization programmatically creates, snapshots, deletes, andrestores software-based virtual networks prior to delivery of productionenvironments.

With network virtualization, the functional equivalent of a networkhypervisor reproduces the complete set of Layer 2 through Layer 7networking services (for example, layers include the switch, routing,access control, firewall, and QoS). As a result, these services may beprogrammatically assembled using the required networking combination toproduce unique, isolated virtual networks for use by each virtualizednetwork 1-n.

In accordance with aspects of the present disclosure, SDN softwareimplements three separate but integrated planes: management, control,and data. The three planes are implemented as a set of processes,modules, and agents residing on three types of nodes: manager,controller, and transport. Transport nodes are the application hostinstances of Hypervisor itself and the edge VM appliances on edge server106.

The API services for SDN are on SDN Manager, providing natural isolationand protection. No SDN management API instances exist on the Applicationserver or Edge server. Programmatic or malicious user attacks could onlybe launched from a Management server. Furthermore, the SDN controllersalso exist on the Management server to host the central Control Planecluster daemons.

The three planes are briefly explained in the following table, followedby more detailed information about the purpose of each:

Plane Description Management This is where administrators interact withthe system to make changes, check logs, and verify configurationsettings. Control This is the mechanism responsible for propagatingchanges implemented at the Management Plane level to the necessarycomponents impacted by the configuration change. Consisting of onlyproprietary API commands, the Control Plane talks directly thehypervisor agents to manipulate data flow and monitoring. Data Thiscontext is where all VM data traffic flows. For the purposes of thedisclosed computing platform, this is where all enclave traffic exists.This includes all East/West and North/South data flow inside and out ofthe enclave. The dedicated encrypted tunnel ensures that the datatraffic from each enclave remains separated.

By design, the traffic on all three planes is completely isolated fromeach other.

The Management Plane may always be limited to the VMs in managementservers 102. As shown in FIG. 13 , RBAC management tools, management,SDN Admin console, and other related management VMs are located on thismanagement cluster, leveraging separate physical resources for compute,storage, networking interfaces, and even VLAN/IP space.

Control plane traffic may be similarly disconnected from the ManagementPlane 1301 via isolated networking interfaces, VLAN. IP space, and APIand kernel calls. The Management Plane 1301 communicates with theControl Plane 1302 by directly interacting with the management hostmanagement ports.

The combination of the Management Plane 1301 and Control Plane 1302dictate the transport nodes’ local Control Plane daemons and forwardingengines.

In one aspect, the Management Plane 1301 may provide a single API entrypoint to computing platform 100, persists administrative userconfiguration, handles user queries, and performs operational tasks onall Management, Control, and Data Plane nodes in the system. The SDNManagement Plane may include the SDN administration page, which is thesingle interface for all user interaction with the system. In computingplatform 100, all configuration settings for the dedicated encryptedtunnel may be pre-determined, making interaction with the SDNadministration page minimal. Minimal interaction means a reduced needfor advanced cloud and infrastructure services training.

The separation of the Management Plane 1301 from the Control Plane 1302may ensure that even in the event of a Management Plane failure, theconfiguration settings disseminated to the application hosts remain ineffect and network communications continue. However, no changes to rulesor traffic flow are allowed while the Management Plane 1301 is down.

Tasks achieved by the Management Plane 1301 may include configurationpersistence (desired logical state), input validation, usermanagement—role assignments, policy management, and background tasktracking.

The primary purpose of the Control Plane is to disseminate configurationsettings throughout computing platform 100 that were implemented in theManagement Plane 1301. This is pre-configured during initial setup. TheControl Plane traffic may include secure proprietary’ API calls madefrom the Admin web page to the Control Plane Hypervisor servers (inother words, the hypervisors participating in the SDN environment). Theclustered controllers may be responsible for maintaining the integrityof the database in the event of a failure.

The Control Plane is split into two parts in SDN: the Central ControlPlane (CCP) 1302, which runs on the SDN Controller cluster nodes, andthe Local Control Plane (LCP) 1303, which runs on the transport nodes(hypervisor server) adjacent to the Data Plane it controls. The CCP 1302may be configured to compute an ephemeral runtime state based onconfiguration from the Management Plane and disseminates informationreported by the Data Plane elements via the local Control Plane. The LCP1303 monitors local link status, computes most ephemeral runtime statebased on updates from the Data Plane 1304 and CCP 1302, and pushesstateless configuration to forwarding engines.

The Data Plane 1304 may include all network data traffic forintra-network communications. On computing platform 100, the Data Plane1304 performs stateless forwarding/transformation of packets based ontables populated by the Control Plane, reports topology information tothe Control Plane, and maintains packet level statistics. The Data Plane1304 is the source of truth for the physical topology and status, suchas VIF (virtual interface) location, or tunnel status. Packets move fromone place to another in the Data Plane 1304. The Data Plane 1304 alsomaintains the status of packets and handles failover between multiplelinks/tunnels.

The Data Plane 1304 is not necessarily fully contained in kernels,drivers, user spaces, or even specific user space processes. The DataPlane 1304 may be constrained to totally stateless forwarding based ontables/rules populated by the Control Plane. The Data Plane 1304 mayalso have components that maintain some amount of state for featuressuch as TCP termination. The state managed by the Control Plane (such asMAC: IP tunnel mappings) is about how to forward the packets; whereasthe state managed by the Data Plane is limited to how to manipulatepayloads.

In one embodiment, computing platform 100 simplifies data encryption byenforcing encryption and authentication policies on microsegments, thusrendering network traffic sniffing useless to an attacker. This featureprovides end-to-end encryption of data, including network transit. Keysare used per microsegment and isolated in the hypervisor, which removesthe complexity of key management from the security administrator.

Encryption rules contain instructions that determine what to do withindividual network packets based on packet properties: authenticate andencrypt, authenticate and decrypt, or only authenticate the packet.Computing platform 100 may utilize a setting that requires allintra-network traffic to be encrypted. If the packet cannot be encryptedbetween East/West hosts, it is dropped, and traffic is terminated.

In accordance with aspects of the present disclosure, the SDN softwarefirewall may be one of multiple subsystems that isolate traffic, and thesoftware firewall provides a layer of guaranteed separation. Thisfirewall is beyond a simple perimeter - it does not allow traffic of anykind to move outside its originating network. Computing platform 100 mayuse micro-segmentation policies as an additional layer of protection andnetwork isolation.

When a network is provisioned for a workload, security is persistentlyenforced regardless of changes in the disclosed computing platformenvironment. This is essential, as platform topologies may change.Networks can be re-deployed, enclave server pools can be expanded, andworkloads can be moved. The one constant in the face of all this changeis the network VMs, along with the need for secured separation.

Computing platform 100′s implementation of the software firewallpolicies ensures a consistent application of network level separationregardless of the changing environment. It is important that thesepolicies be consistently enforced regardless of IP addresses, port,protocol, OS level names, or other operational system-based changes.This persistent security is required, even considering changes of activeor inactive virtualized networks 1-n.

Micro-segmentation gives administrators more useful ways to describe theworkload. Instead of relying merely on IP addresses, administrators maydescribe the inherent characteristics of the workload, such as: type(web, application, or database), use (development, staging, production),and data types (low-sensitivity, financial, personally identifiable).Computing platform 100 may combine these characteristics to defineinherited policy attributes. Thus, a workload associated with onenetwork can never communicate with other networks.

Micro-segmentation embeds security functions into computing platform100′s infrastructure itself. As such, administrators may rely on theavailability of security functions for the broadest spectrum ofworkloads running on computing platform 100. Micro-segmentation speaksto the distributed implementation of the software firewall capability.The rules are applied at the VM object, not at any particular network ordata path. Any rule applied to a VM, regardless of the workload or guestoperating system, applies the rules uniformly irrespective of thelocation of the VM. For example, a rule that restricts certain trafficfrom reaching a VM will continue to function as expected even if the VMis moved from host to host or interface changes are made to theunderlying system.

Micro-segmentation enables computing platform 100 administrators toextend capabilities by integrating additional security functions intotheir portfolio of defense. For instance, administrators might beginwith stateful firewalling distributed throughout the data center but addnext-gen firewall and an intrusion prevention system (IPS) for deepertraffic visibility or agentless anti-malware for better server security.Additionally, these functions cooperate to provide more effectivesecurity than if they were deployed in silos. Micro-segmentation sharesintelligence between security functions, making it possible for thesecurity infrastructure to act concertedly to tailor responses to uniquesituations.

Referring now back to FIG. 4 , the SDN software firewall may be adistributed firewall spread over the Hypervisor host, in accordance withaspects of the present disclosure. The software firewall may enforce thesecurity of virtualized networks traffic. Since each packet leaving anetwork VM is inspected by the firewall, the dedicated encrypted tunnelin computing platform 100 is able to keep network data secure andseparated. The software firewall runs as a kernel service inside theHypervisor host.

With the SDN virtualized network software firewall, a stateful firewallservice for VMs is enforced. The enforcement point is at the VM virtualNIC (vNIC). Every packet that leaves the VM (before VXLAN Tunnel EndPoint (VTEP) encapsulation) or enters the VM (after VTEPde-encapsulation) can be inspected with a firewall policy.

The software firewall rules may be based on Layer 2 up to Layer 4 ofopen systems interconnection model (OSI model). With three-party vendorintegration, SDN can implement security features up to and includingLayer 7. Layer 2 rules are based on MAC address Layer 2 protocols, suchas ARP, RARP and LLDP. Layer 3 rules are based on IP source destination,and Layer 4 uses a TCP or UDP service port.

The software firewall rules policy may be created at a central locationin the hypervisor management server using the management web client. Theobjects are used from the management inventory. As more Hypervisor hostsare added to a hypervisor cluster, the software firewall throughputcapacity is increased.

The above description of the disclosure is provided to enable a personskilled in the art to make or use the disclosure. Various modificationsto the disclosure will be readily apparent to those skilled in the art,and the common principles defined herein may be applied to othervariations without departing from the spirit or scope of the disclosure.For example, the description above may apply to a laminated glazing awell as a single glass substrate.

Furthermore, although elements of the described aspects and/orembodiments may be described or claimed in the singular, the plural iscontemplated unless limitation to the singular is explicitly stated.Additionally, all or a portion of any aspect and or embodiment may beutilized with all or a portion of any other aspect and or embodiment,unless stated otherwise. Thus, the disclosure is not to be limited tothe examples and designs described herein but is to be accorded thewidest scope consistent with the principles and novel features disclosedherein.

1. A system, comprising: a first server computing device, comprising: afirst non-transitory computer-readable storage medium configured tostore a first set of instructions and application data relating to thesystem, and a first processor coupled to the first non-transitorycomputer-readable storage medium and configured to control a firstplurality of modules to execute the first set of instructions forsimultaneously establishing a plurality of logically separate and securenetworks within a self-supported computing environment; a second servercomputing device, comprising: a second non-transitory computer-readablestorage medium configured to store a second set of instructions, and asecond processor coupled to the second non-transitory computer-readablestorage medium and configured to control a second plurality of modulesto execute the second set of instructions for performing out-of-bandmanagement of the system; and a third server computing device,comprising: a third non-transitory computer-readable storage mediumconfigured to store a third set of instructions, and a third processorcoupled to the third non-transitory computer-readable storage medium andconfigured to control a third plurality of modules to execute the thirdset of instructions for controlling inbound and outbound data traffic ofthe plurality of logically separate and secure networks, wherein thesystem is scalable by at least adding additional one or more firstserver computing devices to host additional application data within theself-supported computing environment’s secure configuration and logicalseparation of networks while maintaining the second and third servercomputing devices.